Social behaviors are critical for animals to survive and reproduce. While many social behaviors are innate, they must also be dynamic and flexible to allow adaptation to a rapidly changing environment. The study of social behaviors in model organisms typically requires accurate detection and quantification of such behaviors. Although automated systems for behavioral scoring in some animal species are available, they are generally limited to single animal assays, and their capabilities are restricted either to simple tracking, or to specific behaviors that are measured using a dedicated apparatus (e.g., to measure freezing during fear conditioning, etc.). However, there is increasing interest in quantifying social behaviors in rodents and other animal species, to study mechanisms and treatments for human psychiatric disorders that affect social interactions. In contrast to single animal behaviors, social behaviors are typically scored manually. This is slow, highly labor intensive and subjective, resulting in analysis bottlenecks as well as inconsistencies between different human observers. The issues associated with having humans attempt to manually score behaviors captured in video sequences is viewed by many as limiting progress toward understanding the function of neural circuits and genes controlling social behaviors, and their dysfunction in disorders such as autism.